Pose Estimator für Gradient Boost Classifier¶
Dieser Teil der Projektarbeit befasst sich mit einer anderen Variante für die Klassifizierung der Tennis-Schlägen (Vorhand, Rückhand und Service). Mithilfe des MediaPipe Human Pose Estimation Model sollen zuerst relevante Körperteilpositionen ermittelt und dann anschliessend anhand von Koordinatenpunkten dieser sogenannten Key-Points ein Gradient Boost Classifier trainiert werden. Das Ziel soll sein, dass ein generisches und leichtgewichtiges Klassifizierungmodell erstellt werden kann, welches dann nur noch Koordinaten von Key-Points als Input benötigt und dann mit einem beliebigen Pose Estimation Modell vorgeschalten werden kann.
Aus Zeitgründen wurde nur mit dem MediaPipe Pose Estimation Modell gearbeitet. Enstprechend wurde das ganze Modell dann auf dieses optimiert.
In [1]:
import random
from pathlib import Path
from typing import Callable
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PowerTransformer
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import accuracy_score, f1_score, log_loss, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import mediapipe as mp
import cv2
In [66]:
# load random image
DATA_PATH = Path(
r"C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped")
DATA_PATH_CUSTOM_PICTURES = Path(
r"C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel"
)
all_file_paths = []
for path in DATA_PATH.glob("**/*"):
if path.is_file() and path.suffix == ".jpeg":
all_file_paths.append(path)
all_file_paths_custom_pictures = []
for path in DATA_PATH_CUSTOM_PICTURES.glob("**/*"):
if path.is_file() and path.suffix == ".jpg":
all_file_paths_custom_pictures.append(path)
print(f"total files: {len(all_file_paths)}")
random_file_path = random.choice(all_file_paths)
print(f"random file chosen {random_file_path}")
random_image = cv2.imread(str(random_file_path))
random_image_width = random_image.shape[1]
random_image_height = random_image.shape[0]
fig, ax = plt.subplots()
ax.set_axis_off()
ax.imshow(random_image[..., ::-1])
total files: 705 random file chosen C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_033\F_033_000.jpeg
Out[66]:
<matplotlib.image.AxesImage at 0x1b71f63e000>
Pose Estimation¶
Zuerst sollen die für uns relevanten Körperteile lokalisiert werden. Jeweils die Koordinatenpunkte jedes Key-Points wird dann als Feature weiterverwendet.
In [8]:
def plot_key_points(ax, pose_model, image, image_name):
image_width = image.shape[1]
image_height = image.shape[0]
# mp_drawing = mp.solutions.drawing_utils
# annotated_img = image.copy()
# Processes the image.
results = pose_model.process(image)
# Set the circle radius for drawing the 'landmarks'.
# The radius is scaled as a percentage of the image's height.
# circle_radius = int(.007 * image_height)
# Specifies the drawing style for the 'landmarks'.
# point_spec = mp_drawing.DrawingSpec(color=(220, 100, 0), thickness=-1, circle_radius=circle_radius)
# Draws the 'landmarks' on the image.
# mp_drawing.draw_landmarks(annotated_img,
# landmark_list=results.pose_landmarks,
# landmark_drawing_spec=point_spec)
# Select the coordinates of the points of interest.
l_shoulder_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER].x * image_width)
l_shoulder_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER].y * image_height)
l_elbow_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_ELBOW].x * image_width)
l_elbow_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_ELBOW].y * image_height)
l_wrist_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_WRIST].x * image_width)
l_wrist_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_WRIST].y * image_height)
l_hip_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP].x * image_width)
l_hip_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP].y * image_height)
r_shoulder_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_SHOULDER].x * image_width)
r_shoulder_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_SHOULDER].y * image_height)
r_elbow_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_ELBOW].x * image_width)
r_elbow_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_ELBOW].y * image_height)
r_wrist_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_WRIST].x * image_width)
r_wrist_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_WRIST].y * image_height)
r_hip_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP].x * image_width)
r_hip_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP].y * image_height)
l_ear_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_EAR].x * image_width)
l_ear_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_EAR].y * image_height)
r_ear_x = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_EAR].x * image_width)
r_ear_y = int(results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_EAR].y * image_height)
ax.set_axis_off()
ax.imshow(image[:, :, ::-1])
# left side marked red
ax.plot([l_shoulder_x, l_elbow_x, l_wrist_x, l_hip_x, l_ear_x],
[l_shoulder_y, l_elbow_y, l_wrist_y, l_hip_y, l_ear_y], 'ro')
# right side marked blue
ax.plot([r_shoulder_x, r_elbow_x, r_wrist_x, r_hip_x, r_ear_x],
[r_shoulder_y, r_elbow_y, r_wrist_y, r_hip_y, r_ear_y], 'bo')
ax.set_title(image_name)
In [117]:
mp_pose = mp.solutions.pose
In [15]:
n_rows = 3
n_cols = 5
random_file_paths = random.sample(all_file_paths, n_rows * n_cols)
fig, axs = plt.subplots(n_rows, n_cols, figsize=(12, 8))
with mp_pose.Pose(static_image_mode=True) as pose_model:
for index, random_file_path in enumerate(random_file_paths):
row_index = index // n_cols
col_index = index % n_cols
ax = axs[row_index][col_index]
image_name = random_file_path.stem
image = cv2.imread(str(random_file_path))
plot_key_points(ax, pose_model, image, image_name)
fig.tight_layout()
plt.show()
Bedeutung der Punkte:
- Blau: rechte Seite
- Rot: linke Seite
Wie man sehen kann, funktioniert dieser Pose Estimator laut Stichproben ziemlich gut. Insbesondere werden hauptsächlich Hüft- und Schulterpositionen sehr gut erkannt. Diese sind auch laut unseren Annahmen die wichtigsten Körperteile für die Identifikation von Tennis-Schlägen. Welche Features tatsächlich vom anschliessend zu trainierenden Classifier als wichtig erachtet werden, werden wir später noch sehen.
Wichtig ist noch zu erwähnen, dass der Pose Estimator für manche Bilder keine Körperpositionen ermitteln konnte (ca. 4%).
Achtung: Die Korrektheit der Modellvorhersagen wurde natürlich im Rahmen der Projektarbeit nicht bis aufs letzte untersucht. Lediglich durch Stichproben wurden die festgestellten Körperteilpositionen grob untersucht. Anhand dieser Stichproben wir das von MediaPipe bereitgestellte Modell als gut genug betrachtet um weiterführende Analysen vorzunehmen.
Definition der Features¶
Die Positionen folgender Körperteile werden für den Classifier verwendet (jeweils links und rechts):
- Hüfte
- Schulter
- Ellbogen
- Handgelenk
Der Output des MediaPipe Pose Estimators sind X- und Y-Koordinaten, welche auf die Bildbreite resp. Bildhöhe normiert sind (also Werte zwischen 0 und 1).
Weiterhin werden dann nicht die vom Modell direkt ausgegebenen Koordinaten pro Key-Point (jeweils in X- und Y-Richtung) verwendet, sondern die relative Position der Extremitäten zur Hüftmitte. Der Grund dafür ist, dass sich die Person ja nicht immer an der exakt selben Position im Bild befinden kann.
Dies führt zu folgenden Zahlenwerten:
- X-Achse: Ist der Wert negativ, befindet sich der Punkt auf der linken Seite des Hüft-Zentrums
- Y-Achse: Ist der Wert negativ, befindet sich der Punkt unterhalb des Hüft-Zentrums
In [27]:
feature_demo_file_path = DATA_PATH.joinpath("forehand/seq_012/F_012_003.jpeg")
fig, ax = plt.subplots()
with mp_pose.Pose(static_image_mode=True) as pose_model:
image = cv2.imread(str(feature_demo_file_path))
plot_key_points(ax, pose_model, image, feature_demo_file_path.stem)
Feature Extraktion¶
Die Methoden unten werden die Features für jedes einzelne Bild extrahieren.
Daraus resultieren folgende Features, welche dann als Input für den Classifier verwendet werden sollen:
- Position der rechten und linken Schulter relativ zum Hüftmittelpunkt
- Position des rechten und linken Ellbogens relativ zum Hüftmittelpunkt
- Posision des rechten und linken Handgelenks relativ zum Hüftmittelpunkt
Zusätzlich wird noch folgendes Feature dazu genommen, da es vielleicht noch Informationen enthalten könnte:
- Sequenz Index (da urprünglich die Idee war, aus einer gesamten Bildsequenz einen Tennis-Schlag zu erkannen und wir jetzt aber nur für jedes einzelne Bild eine Vorhersage treffen, wird diese Information auch noch ans Modell weitergegeben. Index 0: Beginn des Schlages, Index 5: Nach dem Schlag)
Weiterhin enthalten im Dataframe, aber nicht als Input für das Modell verwendet werden als Metadaten noch:
- Sequenz Identifier (Die Bilder wurden so abgelegt, dass eine Sequenz aus 5 Bildern besteht. Jede Sequenz ist dann noch über eine Nummer identifizierbar)
- Label
- Pfad der Bilddatei
In [67]:
def get_pose_features_of_image(pose_estimator, image_path: Path, data_augmentation_function: Callable | None = None) -> \
dict[str, float]:
image = cv2.imread(str(image_path))
if data_augmentation_function is not None:
image = data_augmentation_function(image)
results = pose_estimator.process(image)
if results.pose_landmarks is None:
print(f"FAILED FOR IMAGE {image_path}")
return {"relLeftShoulderX": np.nan,
"relLeftShoulderY": np.nan,
"relLeftElbowX": np.nan,
"relLeftElbowY": np.nan,
"relLeftWristX": np.nan,
"relLeftWristY": np.nan,
"relRightShoulderX": np.nan,
"relRightShoulderY": np.nan,
"relRightElbowX": np.nan,
"relRightElbowY": np.nan,
"relRightWristX": np.nan,
"relRightWristY": np.nan}
l_hip_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP].x
l_hip_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP].y
r_hip_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP].x
r_hip_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP].y
center_hip_x = (l_hip_x + r_hip_x) / 2
center_hip_y = (l_hip_y + r_hip_y) / 2
l_shoulder_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER].x
l_shoulder_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER].y
l_elbow_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_ELBOW].x
l_elbow_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_ELBOW].y
l_wrist_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_WRIST].x
l_wrist_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_WRIST].y
r_shoulder_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_SHOULDER].x
r_shoulder_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_SHOULDER].y
r_elbow_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_ELBOW].x
r_elbow_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_ELBOW].y
r_wrist_x = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_WRIST].x
r_wrist_y = results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_WRIST].y
return {"relLeftShoulderX": l_shoulder_x - center_hip_x,
"relLeftShoulderY": l_shoulder_y - center_hip_y,
"relLeftElbowX": l_elbow_x - center_hip_x,
"relLeftElbowY": l_elbow_y - center_hip_y,
"relLeftWristX": l_wrist_x - center_hip_x,
"relLeftWristY": l_wrist_y - center_hip_y,
"relRightShoulderX": r_shoulder_x - center_hip_x,
"relRightShoulderY": r_shoulder_y - center_hip_y,
"relRightElbowX": r_elbow_x - center_hip_x,
"relRightElbowY": r_elbow_y - center_hip_y,
"relRightWristX": r_wrist_x - center_hip_x,
"relRightWristY": r_wrist_y - center_hip_y}
def get_pose_features(pose_estimator, folder_path: Path, initial_dataframe: pd.DataFrame | None = None,
image_pattern: str = "*.jpeg",
data_augmentation_function: Callable | None = None) -> pd.DataFrame:
label = folder_path.name
features = []
for sequence_folder in folder_path.iterdir():
if not sequence_folder.is_dir():
continue
sequence_identifier = int(sequence_folder.name.split("_")[1])
for image_path in sequence_folder.glob(image_pattern):
index_in_sequence = int(image_path.stem.split("_")[2])
image_features = get_pose_features_of_image(pose_estimator, image_path, data_augmentation_function)
image_features.update({"sequenceIdentifier": sequence_identifier,
"indexInSequence": index_in_sequence,
"label": label,
"filePath": str(image_path)})
features.append(image_features)
new_pose_features = pd.DataFrame(features)
if initial_dataframe is not None:
return pd.concat([initial_dataframe, new_pose_features]).reset_index(drop=True)
return new_pose_features
Zuerst werden nur die Features des Online-Datensatzes extrahiert.
In [34]:
with mp_pose.Pose(static_image_mode=True) as pose_estimator:
pose_features = get_pose_features(pose_estimator, DATA_PATH.joinpath("backhand"))
pose_features = get_pose_features(pose_estimator, DATA_PATH.joinpath("forehand"), initial_dataframe=pose_features)
pose_features = get_pose_features(pose_estimator, DATA_PATH.joinpath("serve"), initial_dataframe=pose_features)
pose_features.sample(10)
C:\Git\fhnw-mas-data-science\computer-vision-project-tennis\.venv\Lib\site-packages\google\protobuf\symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_020\B_020_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_048\B_048_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_014\F_014_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_002\S_002_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_005\S_005_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_009\S_009_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_010\S_010_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_014\S_014_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_020\S_020_001.jpeg
Out[34]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | sequenceIdentifier | indexInSequence | label | filePath | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 590 | -0.036711 | -0.178733 | -0.071808 | -0.258573 | -0.111427 | -0.350373 | -0.142644 | -0.122121 | -0.227864 | -0.079792 | -0.240394 | -0.124892 | 9 | 0 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 553 | -0.125156 | -0.165572 | -0.093729 | -0.230305 | -0.233021 | -0.245516 | -0.131296 | -0.172637 | -0.155649 | -0.282426 | -0.285609 | -0.250969 | 1 | 3 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 453 | 0.013918 | -0.163398 | -0.091767 | -0.151382 | -0.022075 | -0.172008 | 0.173835 | -0.112523 | 0.195223 | -0.029394 | 0.316851 | 0.006615 | 29 | 3 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 499 | -0.013379 | -0.137279 | -0.121615 | -0.106553 | -0.123212 | -0.126957 | 0.197747 | -0.123380 | 0.137174 | -0.113207 | 0.050104 | -0.135798 | 38 | 4 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 569 | -0.189294 | -0.079659 | -0.130739 | -0.019889 | -0.065501 | -0.005417 | -0.035150 | -0.104759 | 0.041574 | -0.050702 | 0.012412 | -0.020693 | 4 | 4 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 561 | -0.178581 | -0.085045 | -0.307126 | -0.059908 | -0.341252 | -0.123603 | -0.099816 | -0.138256 | -0.113510 | -0.213358 | -0.159011 | -0.292597 | 3 | 1 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 243 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 48 | 3 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 502 | 0.148860 | -0.116536 | 0.267731 | -0.108356 | 0.365897 | -0.104764 | 0.159207 | -0.107585 | 0.111141 | -0.049659 | 0.129644 | 0.002131 | 39 | 2 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 678 | -0.136193 | -0.068735 | -0.029914 | -0.051168 | 0.062786 | -0.052216 | -0.118083 | -0.114503 | -0.068380 | -0.169608 | -0.021591 | -0.221506 | 26 | 3 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 451 | 0.080941 | -0.087486 | 0.168402 | -0.068214 | 0.259667 | -0.053580 | 0.111792 | -0.083951 | 0.206126 | -0.073190 | 0.307153 | -0.072895 | 29 | 1 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
In [35]:
# Write to pickle to not always have to recalculate
pose_features.to_pickle("pose_features.pkl")
Dann werden die von Gabriel Guerra selbst erstellten Bilder weiterverarbeitet und die Features separat abgespeichert.
In [36]:
with mp_pose.Pose(static_image_mode=True) as pose_estimator:
pose_features_custom_pictures = get_pose_features(pose_estimator, DATA_PATH_CUSTOM_PICTURES.joinpath("backhand"),
image_pattern="*.jpg")
pose_features_custom_pictures = get_pose_features(pose_estimator, DATA_PATH_CUSTOM_PICTURES.joinpath("forehand"),
initial_dataframe=pose_features_custom_pictures,
image_pattern="*.jpg")
pose_features_custom_pictures = get_pose_features(pose_estimator, DATA_PATH_CUSTOM_PICTURES.joinpath("serve"),
initial_dataframe=pose_features_custom_pictures,
image_pattern="*.jpg")
pose_features_custom_pictures.sample(10)
C:\Git\fhnw-mas-data-science\computer-vision-project-tennis\.venv\Lib\site-packages\google\protobuf\symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_000\S_000_003.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_005\S_005_002.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_007\S_007_002.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_020\S_020_002.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_022\S_022_002.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_025\S_025_003.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_030\S_030_003.jpg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_taken_Gabriel\serve\seq_041\S_041_001.jpg
Out[36]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | sequenceIdentifier | indexInSequence | label | filePath | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 395 | -0.061128 | -0.118467 | -0.011548 | -0.079767 | 0.019472 | -0.120845 | -0.037251 | -0.120702 | 0.009215 | -0.057561 | 0.082778 | -0.015521 | 1 | 0 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 422 | -0.064673 | -0.116640 | -0.052254 | -0.159063 | -0.030394 | -0.201577 | -0.092612 | -0.099517 | -0.109717 | -0.097457 | -0.102622 | -0.152977 | 6 | 2 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 11 | -0.052020 | -0.089168 | -0.045211 | -0.031731 | -0.065927 | -0.014396 | -0.024610 | -0.105204 | -0.065167 | -0.058495 | -0.089621 | -0.032091 | 2 | 1 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 85 | -0.034171 | -0.115906 | -0.063988 | -0.059149 | -0.129487 | -0.058093 | 0.024788 | -0.114749 | -0.005622 | -0.063127 | -0.078654 | -0.047009 | 17 | 0 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 97 | -0.002333 | -0.087302 | 0.059527 | -0.043722 | 0.004771 | -0.001491 | -0.057345 | -0.086122 | -0.059929 | -0.049821 | -0.104445 | -0.026191 | 19 | 2 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 17 | -0.125122 | -0.071548 | -0.096722 | -0.029961 | -0.115783 | 0.011625 | -0.094861 | -0.068736 | -0.051477 | -0.035691 | -0.042448 | 0.001814 | 3 | 2 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 405 | -0.046172 | -0.130628 | -0.014088 | -0.162417 | 0.011904 | -0.201144 | -0.079386 | -0.117299 | -0.075878 | -0.063975 | 0.004225 | -0.024320 | 3 | 0 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 394 | -0.156511 | -0.065054 | -0.156299 | -0.014998 | -0.137207 | -0.047052 | -0.066861 | -0.116502 | -0.022670 | -0.147060 | -0.023771 | -0.177139 | 0 | 4 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 364 | -0.051127 | -0.126617 | -0.098081 | -0.076411 | -0.130599 | -0.109491 | 0.051783 | -0.139193 | 0.002591 | -0.125697 | -0.068598 | -0.148067 | 21 | 4 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
| 137 | 0.001696 | -0.088295 | 0.037087 | -0.045065 | -0.019160 | -0.009632 | -0.017612 | -0.090133 | -0.027052 | -0.052251 | -0.066668 | -0.032803 | 27 | 2 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weit... |
In [37]:
pose_features_custom_pictures.to_pickle("pose_features_custom_pictures.pkl")
Modelling¶
Datenbereinigung¶
Wie wir vorher sehen konnten, konnten einige Bilder nicht vom Pose Estimator nicht prozessiert werden. Diese werden nun einfach aus der Tabelle entfernt.
Folgende Tasks wurden bei der Datenbereinigung gemacht:
- Labels zu Klassennummern mappen
- Jede Zeile entfernen, die nur NaN Werte enthält
In [38]:
pose_features = pd.read_pickle("pose_features.pkl")
# map class labels to numbers
class_mapping = {"forehand": 1,
"backhand": 2,
"serve": 3}
pose_features_processed = pose_features.copy()
pose_features_processed["label"] = pose_features["label"].map(class_mapping)
pose_features_processed = pose_features_processed.dropna()
pose_features_processed = pose_features_processed.drop("sequenceIdentifier", axis=1)
pose_features_paths = pose_features_processed.pop("filePath")
pose_features_processed.sample(10)
Out[38]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | indexInSequence | label | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 369 | -0.142413 | -0.121726 | -0.185705 | -0.066945 | -0.144735 | -0.108084 | -0.006561 | -0.144450 | -0.063852 | -0.135478 | -0.159113 | -0.148514 | 4 | 1 |
| 473 | -0.079223 | -0.191349 | -0.162446 | -0.155612 | -0.119114 | -0.179630 | 0.077892 | -0.168242 | 0.138991 | -0.091417 | 0.244277 | -0.060432 | 3 | 1 |
| 700 | -0.137519 | -0.099962 | -0.249786 | -0.073908 | -0.317029 | -0.123112 | -0.065059 | -0.144330 | -0.135307 | -0.151173 | -0.233833 | -0.179025 | 0 | 3 |
| 384 | -0.073423 | -0.142054 | -0.148520 | -0.082874 | -0.158837 | -0.139066 | 0.042610 | -0.163889 | -0.049862 | -0.209671 | -0.188526 | -0.253562 | 4 | 1 |
| 345 | 0.041068 | -0.152902 | 0.193751 | -0.176356 | 0.344669 | -0.209334 | 0.053774 | -0.136904 | 0.046182 | -0.072365 | 0.159198 | -0.083238 | 0 | 1 |
| 583 | -0.205312 | -0.166964 | -0.239829 | -0.206929 | -0.300241 | -0.234995 | -0.103218 | -0.215653 | -0.103803 | -0.320449 | -0.242288 | -0.278665 | 3 | 3 |
| 448 | 0.004148 | -0.110291 | -0.084323 | -0.098836 | -0.038766 | -0.109222 | 0.123678 | -0.080865 | 0.143544 | -0.025264 | 0.225364 | 0.005270 | 3 | 1 |
| 665 | -0.182599 | -0.118634 | -0.314135 | -0.080909 | -0.349538 | -0.157757 | -0.103105 | -0.163608 | -0.174570 | -0.216703 | -0.222141 | -0.321018 | 0 | 3 |
| 45 | -0.135452 | -0.180432 | -0.170982 | -0.097768 | -0.277007 | -0.043311 | -0.077644 | -0.179773 | -0.129065 | -0.093106 | -0.243599 | -0.045918 | 0 | 2 |
| 703 | -0.108149 | -0.153096 | -0.057657 | -0.082293 | 0.008848 | -0.047244 | -0.111532 | -0.163135 | -0.170794 | -0.131732 | -0.152931 | -0.138397 | 3 | 3 |
In [39]:
X_train, X_test, y_train, y_test = train_test_split(
pose_features_processed.loc[:, pose_features_processed.columns != "label"], pose_features_processed["label"],
stratify=pose_features_processed["label"], test_size=0.2, random_state=3)
print(f"Training size: {X_train.shape}")
print(f"Testing size: {X_test.shape}")
Training size: (554, 13) Testing size: (139, 13)
In [40]:
fig, ax = plt.subplots()
ax.hist(y_train, label="train")
ax.hist(y_test, label="test")
ax.legend()
Out[40]:
<matplotlib.legend.Legend at 0x1b71fd160c0>
Model: Gradient Boost Classifier¶
Das Gradient Boost Classifier Modell wurde gewählt, da dieser mit sehr wenigen Input-Features arbeiten kann, sehr schnell ist und einfach anzuwenden ist.
In [41]:
params = {
"n_estimators": 100,
"max_depth": 3,
"learning_rate": 0.01,
"verbose": 1,
"subsample": 0.8
}
gbc_model = GradientBoostingClassifier(**params)
gbc_model
Out[41]:
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)
In [42]:
gbc_model.fit(X_train, y_train)
Iter Train Loss OOB Improve Remaining Time
1 1.0486 0.0131 3.52s
2 1.0274 -0.0119 2.11s
3 1.0132 0.0138 1.62s
4 1.0084 0.0489 1.35s
5 0.9858 -0.0225 1.18s
6 0.9837 0.0561 1.07s
7 0.9701 0.0121 0.95s
8 0.9509 -0.0204 0.82s
9 0.9426 0.0286 0.72s
10 0.9266 -0.0029 0.78s
20 0.8221 0.0203 0.60s
30 0.7342 0.0209 0.46s
40 0.6607 0.0216 0.39s
50 0.5890 -0.0235 0.33s
60 0.5359 -0.0091 0.28s
70 0.4838 -0.0017 0.21s
80 0.4469 -0.0123 0.14s
90 0.3995 -0.0242 0.07s
100 0.3635 -0.0253 0.00s
Out[42]:
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)
Score on train set:
In [50]:
gbc_model.score(X_train, y_train)
Out[50]:
0.9602888086642599
In [52]:
accuracies_train = []
for y_pred_train in gbc_model.staged_predict(X_train):
acc = accuracy_score(y_train, y_pred_train)
accuracies_train.append(acc)
accuracies_test = []
for y_pred_test in gbc_model.staged_predict(X_test):
acc = accuracy_score(y_test, y_pred_test)
accuracies_test.append(acc)
In [59]:
loss_values = gbc_model.oob_scores_
fig, ax = plt.subplots()
line1, = ax.plot(loss_values, label="training loss", color="r")
ax.set_ylabel("Loss")
ax.set_xlabel("Iteration")
ax.grid(True)
ax2 = ax.twinx()
line2, = ax2.plot(accuracies_train, label="training accuracy")
line3, = ax2.plot(accuracies_test, label="testing accuracy")
ax2.set_ylabel("Accuracy")
ax2.legend(handles=[line1, line2, line3])
plt.show()
Im vergleich zum Training von neuronalen Nezten wird hier in der X-Achse nicht die Epochen abgebildet, sondern die Anzahl Iterationen resp. die einzelnen Estimators (Bäume) der jeweiligen Iteration.
So wie es aussieht, sättigt sich die Accuracy ziemlich schnell. Das Modell ist lernfähig!
Um zusätzlich noch eine Aussage treffen zu können wird noch die Confusion Matrix auf dem Test-Datensatz dargestellt:
In [60]:
conf_matrix_plot_1 = ConfusionMatrixDisplay.from_estimator(gbc_model, X_test, y_test)
conf_matrix_plot_1.ax_.set_xticklabels(class_mapping.keys())
conf_matrix_plot_1.ax_.set_yticklabels(class_mapping.keys())
Out[60]:
[Text(0, 0, 'forehand'), Text(0, 1, 'backhand'), Text(0, 2, 'serve')]
Ein weiteres sehr nützliches Feature des Gradient Boost Model ist der Insight in die Wichtigkeit der Input-Features (feature importances). So könnte interpretiert werden, anhand welcher Körperteile tatsächlich ein Tennis-Schlag erkennt werden kann.
In [61]:
feature_importance = gbc_model.feature_importances_
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
# plt.subplot(1, 2, 2)
plt.figure(figsize=(8, 18))
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train.keys()[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Variable Importance')
plt.grid(True)
plt.show()
Test auf den selbst erfassten Bildern¶
Jetzt prüfen wir mal unser erstes Modell, wie es auf den selbst erfassten Bildern, die von Gabriel Guerra erstellt wurden, performt. Es wird erwartet, dass dies wahrscheinlich nicht so gut funktionieren wird, da die Aufnahmeposition etwas anders ist, als im Online-Datensatzes. Auch hier wird zuerst dasselbe Preprocessing durchgeführt, wie beim Online-Datensatz.
In [62]:
pose_features_custom_pictures = pd.read_pickle("pose_features_custom_pictures.pkl")
# map class labels to numbers
class_mapping = {"forehand": 1,
"backhand": 2,
"serve": 3}
pose_features_custom_pictures_processed = pose_features_custom_pictures.copy()
pose_features_custom_pictures_processed["label"] = pose_features_custom_pictures["label"].map(class_mapping)
pose_features_custom_pictures_processed = pose_features_custom_pictures_processed.dropna()
pose_features_custom_pictures_processed = pose_features_custom_pictures_processed.drop("sequenceIdentifier", axis=1)
pose_features_custom_pictures_paths = pose_features_custom_pictures_processed.pop("filePath")
pose_features_custom_pictures_processed.sample(10)
Out[62]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | indexInSequence | label | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 179 | -0.119429 | -0.127351 | -0.157639 | -0.063962 | -0.141205 | -0.015395 | 0.041668 | -0.141527 | 0.144710 | -0.123378 | 0.234510 | -0.149361 | 4 | 2 |
| 362 | -0.014164 | -0.134350 | -0.071040 | -0.115386 | -0.026061 | -0.112919 | 0.121516 | -0.097226 | 0.139399 | -0.036479 | 0.215222 | -0.018587 | 2 | 1 |
| 365 | -0.039417 | -0.119751 | -0.068059 | -0.072121 | -0.019331 | -0.068631 | 0.095374 | -0.113330 | 0.156493 | -0.050237 | 0.151738 | -0.062301 | 0 | 1 |
| 36 | 0.016793 | -0.123044 | -0.024627 | -0.073614 | -0.113570 | -0.089231 | 0.016450 | -0.115620 | -0.036011 | -0.074229 | -0.114612 | -0.063736 | 1 | 2 |
| 248 | -0.019699 | -0.094149 | 0.035151 | -0.048403 | -0.000136 | -0.003669 | -0.015550 | -0.097387 | -0.003114 | -0.052887 | -0.015580 | -0.015168 | 3 | 2 |
| 167 | -0.046337 | -0.102732 | 0.003027 | -0.056974 | -0.039874 | -0.012032 | -0.061385 | -0.100171 | -0.064650 | -0.052427 | -0.129440 | -0.030257 | 2 | 2 |
| 494 | -0.147256 | -0.050453 | -0.090323 | -0.017736 | -0.075851 | 0.019018 | -0.103383 | -0.066711 | -0.032706 | -0.034796 | -0.024534 | 0.000882 | 4 | 3 |
| 320 | -0.074333 | -0.129005 | -0.089212 | -0.072530 | -0.031602 | -0.065764 | 0.095643 | -0.125649 | 0.127947 | -0.060700 | 0.057427 | -0.060652 | 0 | 1 |
| 157 | -0.043306 | -0.087871 | -0.001911 | -0.038426 | -0.045469 | 0.001475 | -0.046818 | -0.090784 | -0.036118 | -0.042108 | -0.071930 | -0.010482 | 2 | 2 |
| 155 | -0.041869 | -0.109611 | -0.076422 | -0.061962 | -0.138030 | -0.055487 | -0.005305 | -0.100654 | -0.043744 | -0.055184 | -0.108909 | -0.038545 | 0 | 2 |
In [63]:
X_custom_pictures = pose_features_custom_pictures_processed.loc[:,
pose_features_custom_pictures_processed.columns != "label"]
y_custom_pictures = pose_features_custom_pictures_processed["label"]
gbc_model.score(X_custom_pictures, y_custom_pictures)
Out[63]:
0.7319932998324958
In [64]:
conf_matrix_plot = ConfusionMatrixDisplay.from_estimator(gbc_model, X_custom_pictures, y_custom_pictures)
conf_matrix_plot.ax_.set_xticklabels(class_mapping.keys())
conf_matrix_plot.ax_.set_yticklabels(class_mapping.keys())
Out[64]:
[Text(0, 0, 'forehand'), Text(0, 1, 'backhand'), Text(0, 2, 'serve')]
Zusammenfassung des ersten Versuches¶
Das erste Modell wurde auf dem Online-Datensatz trainiert. Das Modell hat gelernt und performt ziemlich gut auf dem Test-Set desselben Datensatzes (90% Accuracy). Aber auf dem Datensatz, der selbst erstellt wurde, funktioniert es noch nicht so gut. Dies könnte folgende Ursachen haben:
1. Meistens ist nur immer dieselbe Person abgebildet¶
Im Online-Datensatz ist meistens immer dieselbe Person zu sehen. Nur ganz wenige Bilder sind mit anderen Personen enthalten. Das Modell hat sich wahrscheinlich auf die Verhaltensweise einer bestimmten Person spezialisiert und funktioniert daher bei "fremden" Personen nicht gut.
Um dieses Problem zu Lösen wird folgender Lösungsansatz gewählt:
→ Das selbst erstellte Datenset auch für das Training verwenden
2. Unterschiedliche Grösse der Personen¶
Auch wenn die relative Key-Point-Position zur Hüftmitte auf die Bildgröße normalisiert ist, tritt das Problem auf, wenn der Bildausschnitt nicht richtig gewählt wurde. Die Größe der Person im Bild im Vergleich zur Bildgrösse muss unbedingt gleich sein. Wenn dies nicht der Fall ist, wie hier gezeigt:
/Computer%20Vision%20mit%20Deep%20Learning%20Projektarbe/person_size_problem.png)
Für dieses Problem wird folgender Lösungsansatz gewählt:
→ Standardskalierung und Yeo-Johnson Transformation der Koordinatenpunkte
3. Andere Bildqualität und Winkel¶
Die selbst erstellten Bilder haben eine etwas bessere Auflösung im Vergleich zum Online-Datensatz und die abgebildete Person erscheint in einem anderen Winkel.
→ Data augmentation (schwache Rotation, Blurring, Helligkeit)
Regularisierung¶
Data Augmentation¶
Bilder sollen etwas gedreht, geblurred und verdunkelt, resp. verhellt werden. Natürlich muss das Media Pipe Pose Estimation Model nochmals geprüft werden.
In [110]:
random.seed(3)
# Define functions
def random_rotate_image(image, min_angle: float = -10, max_angle: float = 10, angle: float | None = None):
if angle is None:
angle = random.uniform(min_angle, max_angle)
M = cv2.getRotationMatrix2D((image.shape[1] / 2, image.shape[0] / 2), angle, 1.0)
return cv2.warpAffine(image, M, (image.shape[1], image.shape[0]))
def random_quality(image, kernel_size: tuple[int, int] | None = None):
kernel_sizes = [3, 5, 7, 9, 11]
random_kernel = random.choice(kernel_sizes)
if kernel_size is None:
kernel_size = (random_kernel, random_kernel)
return cv2.GaussianBlur(image, kernel_size, 0)
def random_brightness(image, min_brightness: int = -100, max_brightness: int = 100, brightness: int | None = None):
if brightness is None:
brightness = random.randint(min_brightness, max_brightness)
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
h, s, v = cv2.split(hsv)
v = cv2.add(v, brightness)
v[v > 255] = 255
v[v < 0] = 0
final_hsv = cv2.merge((h, s, v))
image = cv2.cvtColor(final_hsv, cv2.COLOR_HSV2BGR)
return image
def random_augment_image(image):
# sometimes use multiple augmentation functions sometimes just one or none
num_functions_to_apply = random.randint(0, 3)
functions = [random_rotate_image, random_quality, random_brightness]
random.shuffle(functions)
for i in range(num_functions_to_apply):
image = functions[i](image)
return image
Testen der Augmentation-Methods auf einem Bild¶
In [77]:
augment_image_sample_path = DATA_PATH.joinpath("forehand/seq_018/F_018_003.jpeg")
augment_image_sample = cv2.imread(str(augment_image_sample_path))
fig, ax = plt.subplots()
ax.set_axis_off()
ax.imshow(augment_image_sample)
plt.show()
In [80]:
rotated_image = random_rotate_image(augment_image_sample, angle=-10)
fig, ax = plt.subplots()
ax.set_axis_off()
ax.imshow(rotated_image)
plt.show()
In [87]:
blurred_image = random_quality(augment_image_sample, kernel_size=(11, 11))
fig, ax = plt.subplots()
ax.set_axis_off()
ax.imshow(blurred_image)
plt.show()
In [109]:
brighned_image = random_brightness(augment_image_sample, brightness=-100)
fig, ax = plt.subplots()
ax.set_axis_off()
ax.imshow(brighned_image)
plt.show()
In [113]:
n_rows = 3
n_cols = 5
all_file_paths_with_custom_pictures = all_file_paths + all_file_paths_custom_pictures
random_file_paths = random.sample(all_file_paths_with_custom_pictures, n_rows * n_cols)
fig, axs = plt.subplots(n_rows, n_cols, figsize=(12, 8))
for index, random_file_path in enumerate(random_file_paths):
row_index = index // n_cols
col_index = index % n_cols
ax = axs[row_index][col_index]
ax.set_axis_off()
image_name = random_file_path.stem
image = cv2.imread(str(random_file_path))
aug_image = random_augment_image(image)
ax.imshow(aug_image)
fig.tight_layout()
plt.show()
Auf jedem Bild werden dann entweder alle, einzelne oder keine Augmentation-Methoden zufällig angewendet. Mal schauen, wie der Pose Estimator damit klar kommt.
In [116]:
n_rows = 3
n_cols = 5
all_file_paths_with_custom_pictures = all_file_paths + all_file_paths_custom_pictures
random_file_paths = random.sample(all_file_paths_with_custom_pictures, n_rows * n_cols)
fig, axs = plt.subplots(n_rows, n_cols, figsize=(12, 8))
with mp_pose.Pose(static_image_mode=True) as pose_model:
for index, random_file_path in enumerate(random_file_paths):
row_index = index // n_cols
col_index = index % n_cols
ax = axs[row_index][col_index]
image_name = random_file_path.stem
image = cv2.imread(str(random_file_path))
image = random_augment_image(image)
plot_key_points(ax, pose_model, image, image_name)
fig.tight_layout()
plt.show()
C:\Git\fhnw-mas-data-science\computer-vision-project-tennis\.venv\Lib\site-packages\google\protobuf\symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
Laut Stichprobenprüfung funktioniert die Key-Point Erkennung immer noch. Nur konnte der Estimator viel mehr Bilder nicht mehr prozessieren.
Erstellung von Features mit zufällig augmentierten Bilder¶
In [118]:
with mp_pose.Pose(static_image_mode=True) as pose_estimator:
pose_features_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("backhand"),
data_augmentation_function=random_augment_image)
pose_features_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("forehand"),
initial_dataframe=pose_features_with_aug,
data_augmentation_function=random_augment_image)
pose_features_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("serve"),
initial_dataframe=pose_features_with_aug,
data_augmentation_function=random_augment_image)
with mp_pose.Pose(static_image_mode=True) as pose_estimator:
pose_features_custom_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("backhand"),
data_augmentation_function=random_augment_image)
pose_features_custom_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("forehand"),
initial_dataframe=pose_features_custom_with_aug,
data_augmentation_function=random_augment_image)
pose_features_custom_with_aug = get_pose_features(pose_estimator, DATA_PATH.joinpath("serve"),
initial_dataframe=pose_features_custom_with_aug,
data_augmentation_function=random_augment_image)
C:\Git\fhnw-mas-data-science\computer-vision-project-tennis\.venv\Lib\site-packages\google\protobuf\symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_001\B_001_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_020\B_020_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_029\B_029_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_033\B_033_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_043\B_043_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_045\B_045_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_048\B_048_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_058\B_058_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_014\F_014_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_034\F_034_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_034\F_034_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_035\F_035_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_038\F_038_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_002\S_002_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_005\S_005_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_006\S_006_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_010\S_010_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_011\S_011_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_014\S_014_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_016\S_016_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_017\S_017_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_022\S_022_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_029\S_029_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_009\B_009_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_020\B_020_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_026\B_026_001.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_044\B_044_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_048\B_048_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_050\B_050_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_054\B_054_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_002\F_002_001.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_014\F_014_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_017\F_017_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_019\F_019_001.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_023\F_023_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_038\F_038_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_002\S_002_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_005\S_005_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_008\S_008_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_009\S_009_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_010\S_010_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_012\S_012_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_012\S_012_004.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_013\S_013_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_015\S_015_001.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_017\S_017_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_018\S_018_003.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_020\S_020_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_020\S_020_001.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_025\S_025_000.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_025\S_025_002.jpeg FAILED FOR IMAGE C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_028\S_028_003.jpeg
In [119]:
pose_features_with_aug.to_pickle("pose_features_with_aug.pkl")
pose_features_custom_with_aug.to_pickle("pose_features_custom_with_aug.pkl")
Aufgrund der Data Augmentation konnte der Pose Estimator ganz viele Bilder nicht mehr prozessieren.
In [ ]:
pose_features_with_aug = pd.read_pickle("pose_features_with_aug.pkl")
pose_features_custom_with_aug = pd.read_pickle("pose_features_custom_with_aug.pkl")
In [120]:
all_pose_features = pd.concat((pose_features_with_aug, pose_features_custom_with_aug)).reset_index(drop=True)
all_pose_features.sample(10)
Out[120]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | sequenceIdentifier | indexInSequence | label | filePath | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1072 | 0.153297 | -0.096289 | 0.248878 | -0.084182 | 0.337902 | -0.060664 | 0.029058 | -0.091699 | -0.040356 | -0.043380 | -0.093412 | 0.004380 | 12 | 2 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_012\F_012_002.jpeg |
| 578 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 6 | 3 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_006\S_006_003.jpeg |
| 1279 | -0.185382 | -0.079843 | -0.139150 | -0.013709 | -0.077641 | -0.033337 | -0.053431 | -0.137050 | 0.035057 | -0.164841 | 0.053413 | -0.166933 | 5 | 4 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_005\S_005_004.jpeg |
| 285 | -0.095441 | -0.172182 | -0.139186 | -0.083108 | -0.277963 | -0.053564 | -0.008860 | -0.178580 | -0.019166 | -0.087037 | -0.119879 | -0.035563 | 57 | 0 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_057\B_057_000.jpeg |
| 1093 | -0.024425 | -0.133092 | 0.091573 | -0.128675 | 0.203474 | -0.138812 | -0.125597 | -0.100600 | -0.152564 | -0.042437 | -0.189509 | -0.003687 | 16 | 3 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_016\F_016_003.jpeg |
| 879 | -0.107256 | -0.236241 | -0.018290 | -0.261552 | 0.038097 | -0.298973 | 0.098681 | -0.226458 | 0.253933 | -0.222952 | 0.268953 | -0.271375 | 34 | 4 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_034\B_034_004.jpeg |
| 863 | -0.153481 | -0.188172 | -0.240547 | -0.133509 | -0.347680 | -0.115203 | 0.020589 | -0.214808 | 0.064998 | -0.124022 | -0.019982 | -0.074770 | 31 | 3 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_031\B_031_003.jpeg |
| 759 | -0.121699 | -0.204760 | -0.028162 | -0.230542 | 0.155640 | -0.260302 | 0.062938 | -0.185188 | 0.186216 | -0.158380 | 0.221797 | -0.232061 | 10 | 4 | backhand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\backhand\seq_010\B_010_004.jpeg |
| 1037 | -0.001574 | -0.108354 | -0.024809 | -0.057011 | -0.032543 | -0.013881 | 0.072442 | -0.116182 | 0.154518 | -0.133151 | 0.259354 | -0.138681 | 5 | 2 | forehand | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\forehand\seq_005\F_005_002.jpeg |
| 675 | -0.039017 | -0.169018 | -0.013879 | -0.232693 | 0.018350 | -0.276607 | -0.133989 | -0.118227 | -0.148962 | -0.034836 | -0.051607 | -0.069640 | 26 | 0 | serve | C:\Users\simu_\OneDrive\Dokumente\Studium Weiterbildung\2023 MAS Data Science FHNW\04_Projektarbeiten\20241102_CAS_Deep_Learning\Tennis_classifier\images_cropped\serve\seq_026\S_026_000.jpeg |
Datenbereinigung¶
In [121]:
# map class labels to numbers
class_mapping = {"forehand": 1,
"backhand": 2,
"serve": 3}
all_pose_features_processed = all_pose_features.copy()
all_pose_features_processed["label"] = pose_features["label"].map(class_mapping)
all_pose_features_processed = all_pose_features_processed.dropna()
all_pose_features_processed = all_pose_features_processed.drop("sequenceIdentifier", axis=1)
all_pose_features_paths = all_pose_features_processed.pop("filePath")
print(f"Number of dropped samples: {len(all_pose_features) - len(all_pose_features_processed)}")
all_pose_features_processed.sample(10)
Number of dropped samples: 732
Out[121]:
| relLeftShoulderX | relLeftShoulderY | relLeftElbowX | relLeftElbowY | relLeftWristX | relLeftWristY | relRightShoulderX | relRightShoulderY | relRightElbowX | relRightElbowY | relRightWristX | relRightWristY | indexInSequence | label | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 70 | -0.133191 | -0.191991 | -0.169328 | -0.089743 | -0.279761 | -0.017907 | -0.063555 | -0.196377 | -0.102798 | -0.093327 | -0.200873 | -0.011109 | 0 | 2.0 |
| 283 | -0.056981 | -0.126166 | -0.042959 | -0.063857 | -0.119239 | -0.013432 | -0.028551 | -0.134859 | -0.056302 | -0.066582 | -0.139329 | -0.016690 | 3 | 2.0 |
| 565 | -0.083431 | -0.160094 | -0.117766 | -0.236756 | -0.184546 | -0.322254 | -0.159230 | -0.111897 | -0.243566 | -0.074888 | -0.294834 | -0.148426 | 0 | 3.0 |
| 191 | -0.088479 | -0.113996 | -0.125113 | -0.063518 | -0.160857 | -0.042806 | 0.045665 | -0.117186 | -0.005452 | -0.061362 | -0.087905 | -0.054061 | 1 | 2.0 |
| 128 | -0.141878 | -0.241048 | -0.155852 | -0.114422 | -0.186201 | -0.009801 | -0.019730 | -0.262879 | -0.100214 | -0.139504 | -0.144922 | -0.042837 | 3 | 2.0 |
| 569 | -0.205421 | -0.073030 | -0.138478 | -0.018310 | -0.074843 | -0.011723 | -0.055318 | -0.105118 | 0.030513 | -0.054327 | 0.004321 | -0.025630 | 4 | 3.0 |
| 45 | -0.135452 | -0.180432 | -0.170982 | -0.097768 | -0.277007 | -0.043311 | -0.077644 | -0.179773 | -0.129065 | -0.093106 | -0.243599 | -0.045918 | 0 | 2.0 |
| 630 | -0.061991 | -0.134354 | -0.098478 | -0.093150 | -0.169807 | -0.096404 | -0.115291 | -0.126031 | -0.164068 | -0.071142 | -0.199295 | -0.049002 | 0 | 3.0 |
| 166 | -0.182214 | -0.170299 | -0.209433 | -0.077066 | -0.306509 | -0.031766 | -0.115684 | -0.176306 | -0.117424 | -0.082560 | -0.237544 | -0.024880 | 1 | 2.0 |
| 216 | -0.163635 | -0.213000 | -0.098482 | -0.119448 | -0.274752 | -0.067591 | -0.113915 | -0.225859 | -0.142976 | -0.122832 | -0.282147 | -0.065184 | 1 | 2.0 |
In [123]:
X_train, X_test, y_train, y_test = train_test_split(
all_pose_features_processed.loc[:, all_pose_features_processed.columns != "label"],
all_pose_features_processed["label"],
stratify=all_pose_features_processed["label"], test_size=0.2, random_state=3)
print(f"Training size: {X_train.shape}")
print(f"Testing size: {X_test.shape}")
Training size: (542, 13) Testing size: (136, 13)
Modelling¶
In [174]:
all_input_channels = [x for x in all_pose_features_processed if not x in ["label"]]
channels_to_power_transform = [x for x in all_pose_features_processed if not x in ["indexInSequence", "label"]]
gbc_params = {
"n_estimators": 100,
"max_depth": 3,
"learning_rate": 0.01,
"verbose": 1,
"subsample": 0.8
}
gbc = GradientBoostingClassifier(**gbc_params)
standard_scaler = ("scaler", StandardScaler(), all_input_channels)
power_transformer = ("yeoJohnson", PowerTransformer(), channels_to_power_transform)
preprocessor = ColumnTransformer(transformers=[
standard_scaler,
power_transformer
], remainder="passthrough")
gbc_pipeline_model = Pipeline([
("preprocessor", preprocessor),
("model", gbc)
])
gbc_pipeline_model
Out[174]:
Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),
('yeoJohnson',
PowerTransformer(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8,
verbose=1))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),
('yeoJohnson',
PowerTransformer(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8,
verbose=1))])ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY',
'indexInSequence']),
('yeoJohnson', PowerTransformer(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY'])])['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY', 'indexInSequence']
StandardScaler()
['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY']
PowerTransformer()
passthrough
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)
In [181]:
# gbc_pipeline_model["preprocessor"].transform(X_train)
gbc_pipeline_model["preprocessor"].transform(X_train)
Out[181]:
array([[-0.8311903 , 0.38693474, -1.30743056, ..., -1.58114907,
-1.00259538, -1.33897336],
[-0.40360735, -0.73008449, -0.92014485, ..., 0.21223179,
-1.47043435, 0.47071971],
[-0.20583744, -1.2619714 , -0.15310577, ..., -0.01214159,
-1.56699933, 0.50835943],
...,
[-0.79577341, 0.51060432, -0.50569297, ..., -0.91010222,
0.35798891, -0.94322194],
[-1.51696534, 0.15521723, -0.85985484, ..., 0.76121065,
-1.42340327, 1.18495504],
[-1.41529247, -0.77571741, -1.13003838, ..., 0.17830372,
-0.81379546, 0.70653998]])
Training
In [175]:
gbc_pipeline_model.fit(X_train, y_train)
Iter Train Loss OOB Improve Remaining Time
1 1.0436 0.0129 1.19s
2 1.0349 0.0284 1.11s
3 1.0213 0.0095 1.12s
4 1.0149 0.0370 1.12s
5 0.9973 -0.0095 1.07s
6 0.9818 0.0011 1.07s
7 0.9804 0.0534 1.07s
8 0.9499 -0.0621 1.05s
9 0.9569 0.0842 1.05s
10 0.9345 -0.0341 1.03s
20 0.8330 -0.0136 0.87s
30 0.7499 -0.0034 0.76s
40 0.6887 0.0048 0.70s
50 0.6266 -0.0081 0.60s
60 0.5702 0.0010 0.48s
70 0.5259 0.0081 0.36s
80 0.4727 -0.0464 0.23s
90 0.4463 0.0033 0.12s
100 0.4137 -0.0156 0.00s
Out[175]:
Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),
('yeoJohnson',
PowerTransformer(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8,
verbose=1))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),
('yeoJohnson',
PowerTransformer(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8,
verbose=1))])ColumnTransformer(remainder='passthrough',
transformers=[('scaler', StandardScaler(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY',
'indexInSequence']),
('yeoJohnson', PowerTransformer(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY'])])['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY', 'indexInSequence']
StandardScaler()
['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY']
PowerTransformer()
[]
passthrough
GradientBoostingClassifier(learning_rate=0.01, subsample=0.8, verbose=1)
In [143]:
gbc_pipeline_model.score(X_test, y_test)
Out[143]:
0.9117647058823529
In [145]:
accuracies_train = []
X_train_transformed = gbc_pipeline_model["preprocessor"].transform(X_train)
for y_pred_train in gbc_pipeline_model["model"].staged_predict(X_train_transformed):
acc = accuracy_score(y_train, y_pred_train)
accuracies_train.append(acc)
accuracies_test = []
X_test_transformed = gbc_pipeline_model["preprocessor"].transform(X_test)
for y_pred_test in gbc_pipeline_model["model"].staged_predict(X_test_transformed):
acc = accuracy_score(y_test, y_pred_test)
accuracies_test.append(acc)
loss_values = gbc_pipeline_model["model"].oob_scores_
fig, ax = plt.subplots()
line1, = ax.plot(loss_values, label="training loss", color="r")
ax.set_ylabel("Loss")
ax.set_xlabel("Iteration")
ax.grid(True)
ax2 = ax.twinx()
line2, = ax2.plot(accuracies_train, label="training accuracy")
line3, = ax2.plot(accuracies_test, label="testing accuracy")
ax2.set_ylabel("Accuracy")
ax2.legend(handles=[line1, line2, line3])
plt.show()
Hyper-Parameter Tuning¶
- Grid-Search on (learning_rate, n_estimators, max_depth, subsample, etc.)
In [148]:
# note: verbosity is listed as hyperparameter to override the model instanciated param verbose=1
param_grid = {
"model__n_estimators": [100, 200, 500, 1000, 2000],
"model__max_depth": [3, 4, 5, 6],
"model__learning_rate": [0.001, 0.01, 0.1],
"model__verbose": [0],
"model__subsample": [0.6, 0.8, 1]
}
gbc_gs_pipeline_model = GridSearchCV(estimator=gbc_pipeline_model,
param_grid=param_grid,
cv=5,
refit=True,
scoring="accuracy",
verbose=4)
gbc_gs_pipeline_model.fit(X_train, y_train)
Fitting 5 folds for each of 180 candidates, totalling 900 fits [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.624 total time= 0.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.569 total time= 1.4s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.593 total time= 0.8s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.630 total time= 0.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.519 total time= 0.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.615 total time= 0.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.596 total time= 0.7s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.583 total time= 0.8s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.639 total time= 0.7s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.537 total time= 0.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.606 total time= 0.9s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.578 total time= 0.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.620 total time= 0.9s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.648 total time= 0.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.556 total time= 0.9s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.651 total time= 1.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.688 total time= 1.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.657 total time= 1.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.694 total time= 1.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.611 total time= 1.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.651 total time= 1.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.697 total time= 1.5s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.676 total time= 1.5s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.685 total time= 1.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.593 total time= 1.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.670 total time= 2.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.661 total time= 1.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.648 total time= 2.1s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.713 total time= 1.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.583 total time= 1.9s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 3.1s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.844 total time= 3.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 3.1s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 3.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.833 total time= 3.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.872 total time= 3.9s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 3.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 4.1s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 4.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.806 total time= 4.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.844 total time= 4.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.844 total time= 4.7s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.870 total time= 4.8s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.824 total time= 4.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.796 total time= 4.8s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 6.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 6.3s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 6.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 6.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 6.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 8.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 7.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 8.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 8.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 8.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.899 total time= 9.7s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.862 total time= 9.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.889 total time= 9.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 9.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.861 total time= 9.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 12.7s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 13.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 13.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 12.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 13.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 16.1s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.853 total time= 16.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 16.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 16.1s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 16.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.899 total time= 19.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.872 total time= 19.8s [CV 3/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.926 total time= 19.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 19.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.880 total time= 19.4s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.624 total time= 0.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.606 total time= 0.8s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.602 total time= 0.8s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.639 total time= 0.8s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.519 total time= 0.8s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.624 total time= 1.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.624 total time= 1.0s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.602 total time= 1.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.639 total time= 1.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.537 total time= 1.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.615 total time= 1.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.661 total time= 1.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.667 total time= 1.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.704 total time= 1.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.556 total time= 1.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.670 total time= 1.7s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.697 total time= 1.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.713 total time= 1.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.704 total time= 1.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.648 total time= 1.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.670 total time= 2.1s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.688 total time= 2.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.713 total time= 2.1s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.722 total time= 2.1s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.676 total time= 2.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.716 total time= 2.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.679 total time= 2.5s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.769 total time= 2.5s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.741 total time= 2.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.713 total time= 2.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 4.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 4.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 4.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 4.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.843 total time= 4.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 5.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 5.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 5.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.833 total time= 5.4s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.899 total time= 6.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.844 total time= 6.4s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 6.4s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.889 total time= 6.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.778 total time= 6.4s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 8.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 8.4s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 8.4s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 8.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 8.5s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 10.5s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 10.5s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 10.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 10.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 10.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.908 total time= 13.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.872 total time= 13.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 14.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 13.5s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.824 total time= 13.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 17.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 16.7s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 16.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 16.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 16.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 21.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.872 total time= 20.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.935 total time= 21.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 21.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 21.4s [CV 1/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 25.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 25.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 25.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 25.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.833 total time= 25.5s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.651 total time= 1.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.642 total time= 1.0s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.611 total time= 1.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.657 total time= 1.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.556 total time= 1.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.651 total time= 1.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.661 total time= 1.3s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.648 total time= 1.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.667 total time= 1.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.574 total time= 1.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.688 total time= 1.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.651 total time= 1.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.685 total time= 1.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.722 total time= 1.5s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.648 total time= 1.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.697 total time= 2.1s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.706 total time= 2.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.741 total time= 2.1s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.722 total time= 2.1s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.685 total time= 2.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.706 total time= 2.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.706 total time= 2.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.731 total time= 2.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.750 total time= 2.7s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.731 total time= 2.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.734 total time= 3.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.716 total time= 3.3s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.722 total time= 3.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.759 total time= 3.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.750 total time= 3.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 5.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 5.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 5.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 5.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 6.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 6.8s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 6.9s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 6.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 6.8s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 8.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.862 total time= 8.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.889 total time= 8.2s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.889 total time= 8.2s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.833 total time= 8.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 10.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 10.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 10.8s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 10.7s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 10.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 13.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 13.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 13.5s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 13.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 13.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.908 total time= 16.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.881 total time= 16.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.880 total time= 16.4s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.889 total time= 16.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.824 total time= 16.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 21.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 21.5s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 21.5s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 21.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 21.5s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 27.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 27.2s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 27.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 27.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 27.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 32.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.890 total time= 32.8s [CV 3/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.880 total time= 32.5s [CV 4/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 32.6s [CV 5/5] END model__learning_rate=0.001, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.824 total time= 33.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.651 total time= 1.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.615 total time= 1.3s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.630 total time= 1.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.667 total time= 1.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.574 total time= 1.3s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.642 total time= 1.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.642 total time= 1.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.639 total time= 1.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.676 total time= 1.7s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.593 total time= 1.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.670 total time= 1.9s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.688 total time= 1.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.685 total time= 2.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.731 total time= 1.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.657 total time= 1.9s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.706 total time= 2.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.716 total time= 2.6s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.741 total time= 2.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.741 total time= 2.7s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.704 total time= 2.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.706 total time= 3.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.734 total time= 3.3s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.731 total time= 3.4s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.787 total time= 3.5s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.713 total time= 3.5s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.743 total time= 4.0s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.697 total time= 4.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.704 total time= 4.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.843 total time= 4.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.750 total time= 4.0s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.6s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.881 total time= 6.7s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.7s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 7.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 6.7s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 8.3s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 8.5s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 8.3s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 8.4s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 8.5s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.927 total time= 10.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.862 total time= 9.9s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 10.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.889 total time= 9.9s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.815 total time= 9.9s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 13.2s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 13.7s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 13.9s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 13.5s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 13.9s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 16.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 17.0s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 17.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 17.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 17.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 20.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.872 total time= 20.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.880 total time= 20.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.907 total time= 20.0s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.815 total time= 20.2s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 26.4s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 27.0s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 26.6s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 27.1s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 27.1s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 33.7s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 34.1s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 33.9s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 35.3s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 34.6s [CV 1/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 40.8s [CV 2/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.881 total time= 40.4s [CV 3/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.889 total time= 40.0s [CV 4/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 40.1s [CV 5/5] END model__learning_rate=0.001, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.843 total time= 40.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 0.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.881 total time= 0.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 0.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 0.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.843 total time= 0.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 0.8s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 0.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 0.8s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 0.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 0.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.899 total time= 0.9s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.862 total time= 0.9s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.889 total time= 0.9s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.898 total time= 0.9s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.870 total time= 0.9s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 1.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.881 total time= 1.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 1.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 1.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.843 total time= 1.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 1.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.872 total time= 1.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 1.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 1.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 1.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.899 total time= 1.9s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.872 total time= 2.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.926 total time= 1.9s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.898 total time= 1.9s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.880 total time= 1.9s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 3.1s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.881 total time= 3.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 3.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 3.3s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 3.1s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 4.1s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 4.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 4.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 4.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.9s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.881 total time= 4.9s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 4.9s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 4.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.861 total time= 4.8s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 6.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 6.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 6.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 6.4s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 6.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 8.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 8.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 8.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 8.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 8.1s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 9.9s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.890 total time= 9.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 9.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 9.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.870 total time= 9.8s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 13.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 12.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 12.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 12.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 12.8s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 16.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 16.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 16.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 16.3s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 16.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.936 total time= 19.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.899 total time= 19.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 19.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 19.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.843 total time= 19.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 0.8s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.872 total time= 0.8s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 0.8s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 0.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 0.8s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 1.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.862 total time= 1.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 1.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 1.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.908 total time= 1.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.881 total time= 1.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 1.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.898 total time= 1.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.824 total time= 1.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 1.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 1.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 1.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 2.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.872 total time= 2.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 2.1s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.908 total time= 2.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.908 total time= 2.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 2.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.907 total time= 2.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.833 total time= 2.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 4.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 4.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 4.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 4.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 4.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 5.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 5.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 5.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.908 total time= 6.4s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 6.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 6.4s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 6.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.843 total time= 6.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 8.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 8.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 8.4s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 8.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 8.5s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 10.5s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 10.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 10.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 10.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 10.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.908 total time= 12.8s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 12.9s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.907 total time= 12.8s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 12.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.870 total time= 13.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 16.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 17.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 17.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 17.3s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 17.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 21.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 21.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 21.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 21.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 21.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 25.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 25.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 25.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 26.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.861 total time= 26.9s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 1.1s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 1.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 1.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 1.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 1.1s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 1.4s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 1.4s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 1.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.908 total time= 1.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.881 total time= 1.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.880 total time= 1.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.889 total time= 1.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.824 total time= 1.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.890 total time= 2.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 2.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 2.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 2.1s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.872 total time= 2.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 2.7s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 2.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 3.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.890 total time= 3.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.880 total time= 3.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.907 total time= 3.4s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.833 total time= 3.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 5.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 5.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 5.3s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 5.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 7.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 7.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 6.8s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.927 total time= 8.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.908 total time= 8.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 8.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 8.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.815 total time= 8.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 10.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 10.2s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 10.4s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 10.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 10.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 13.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 13.4s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 13.2s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 13.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 13.5s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 16.2s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.927 total time= 16.4s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.889 total time= 16.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 16.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.815 total time= 16.4s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 20.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 20.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 20.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 21.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 20.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 26.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 26.5s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 26.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 26.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 26.9s [CV 1/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 32.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 32.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 32.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 33.1s [CV 5/5] END model__learning_rate=0.01, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.861 total time= 32.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 1.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 1.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 1.4s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 1.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.7s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 1.7s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 1.7s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 1.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 1.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.917 total time= 2.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.872 total time= 1.9s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.889 total time= 2.1s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 2.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.815 total time= 2.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 2.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.899 total time= 2.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.6s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 2.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 3.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.881 total time= 3.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 3.4s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 3.3s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 3.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.881 total time= 3.9s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.889 total time= 4.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.898 total time= 4.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.833 total time= 4.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 6.4s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 6.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 6.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 8.3s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 8.3s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 8.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 8.4s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 8.5s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.908 total time= 10.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.890 total time= 10.1s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.880 total time= 9.9s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 9.9s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.806 total time= 10.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 12.6s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 13.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 13.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 13.2s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 12.7s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 16.9s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 16.5s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 16.5s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 16.6s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 16.6s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.899 total time= 20.1s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.881 total time= 20.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 20.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 19.9s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.833 total time= 20.2s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 25.0s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 25.0s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 25.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 25.5s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 25.0s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 32.5s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 32.6s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 32.3s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 32.8s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 32.3s [CV 1/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.899 total time= 30.8s [CV 2/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.890 total time= 26.5s [CV 3/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 30.0s [CV 4/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 31.0s [CV 5/5] END model__learning_rate=0.01, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.824 total time= 30.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 0.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 0.6s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 0.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 0.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 0.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 0.7s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.899 total time= 0.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 0.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 0.8s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 0.7s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.908 total time= 0.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.881 total time= 0.9s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 0.9s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.917 total time= 0.9s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.852 total time= 0.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.2s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 1.2s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 1.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 1.2s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 1.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 1.5s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 1.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 1.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 1.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.899 total time= 1.9s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.889 total time= 1.9s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 1.9s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.843 total time= 1.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 3.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 3.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 3.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 3.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 3.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 4.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 3.9s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 4.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.890 total time= 4.8s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.880 total time= 4.8s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.9s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.843 total time= 4.8s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 5.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 5.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 5.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 5.2s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 6.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.3s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 6.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 6.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.908 total time= 7.8s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.890 total time= 7.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.880 total time= 7.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 7.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.852 total time= 6.4s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 6.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 6.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 6.5s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 6.2s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 7.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 7.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 7.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 7.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.852 total time= 6.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 8.7s [CV 2/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.890 total time= 8.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.880 total time= 8.3s [CV 4/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 8.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=3, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.852 total time= 7.3s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 0.8s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.908 total time= 0.8s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 0.8s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 0.8s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 0.8s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 1.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 1.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 1.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 1.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.908 total time= 1.2s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.899 total time= 1.2s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.917 total time= 1.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 1.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.861 total time= 1.2s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 1.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.6s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 1.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 1.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 2.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 2.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 2.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 2.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 2.4s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.899 total time= 2.5s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 2.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.898 total time= 2.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.852 total time= 2.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 4.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 4.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.889 total time= 4.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 4.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 4.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 4.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 4.8s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 5.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 5.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 4.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.908 total time= 5.4s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.899 total time= 4.8s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 5.4s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 5.4s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.852 total time= 4.4s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.945 total time= 5.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 5.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 4.8s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 5.4s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 5.2s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 5.4s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 5.4s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 4.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 5.8s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.899 total time= 5.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.907 total time= 5.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.907 total time= 5.7s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.843 total time= 4.7s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 6.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.8s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 6.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 5.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.936 total time= 6.4s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 6.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 5.7s [CV 1/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 6.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.899 total time= 6.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 6.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.907 total time= 6.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=4, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.843 total time= 5.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 1.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 1.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 1.3s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.890 total time= 1.3s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 1.3s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 1.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 1.3s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.917 total time= 1.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.917 total time= 1.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 1.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.926 total time= 1.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.824 total time= 1.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.945 total time= 2.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 2.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 2.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 3.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 2.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 2.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 2.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.908 total time= 3.2s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 3.3s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.889 total time= 3.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.898 total time= 3.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.861 total time= 3.4s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 4.2s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 4.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 4.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 4.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.852 total time= 3.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 6.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 4.8s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 4.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.917 total time= 3.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.889 total time= 4.4s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 4.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.861 total time= 3.8s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 6.3s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.1s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.907 total time= 6.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 6.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 5.3s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 5.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 5.2s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.889 total time= 4.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.908 total time= 5.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.889 total time= 5.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.898 total time= 4.9s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.843 total time= 4.3s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 5.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.5s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 6.0s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 6.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 5.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 5.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.9s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.9s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 5.4s [CV 1/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.908 total time= 5.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.917 total time= 4.9s [CV 3/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.889 total time= 5.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 5.8s [CV 5/5] END model__learning_rate=0.1, model__max_depth=5, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.852 total time= 5.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 1.2s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.3s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.898 total time= 1.4s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 1.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.6, model__verbose=0;, score=0.861 total time= 1.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 1.7s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 1.6s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 1.7s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 1.7s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=0.8, model__verbose=0;, score=0.861 total time= 1.7s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.908 total time= 2.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.881 total time= 2.2s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.889 total time= 2.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.907 total time= 2.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=100, model__subsample=1, model__verbose=0;, score=0.852 total time= 2.1s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 2.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.6s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 2.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.944 total time= 2.8s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 2.7s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.936 total time= 3.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 3.5s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 3.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 3.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 3.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.899 total time= 3.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.881 total time= 3.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.898 total time= 3.3s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.917 total time= 3.4s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=200, model__subsample=1, model__verbose=0;, score=0.843 total time= 3.3s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.945 total time= 4.6s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 4.7s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 4.6s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 4.4s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.6, model__verbose=0;, score=0.880 total time= 4.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.927 total time= 4.4s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.908 total time= 4.3s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.907 total time= 4.2s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.935 total time= 4.3s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 4.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.890 total time= 3.5s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.881 total time= 3.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.898 total time= 3.4s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.907 total time= 3.6s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=500, model__subsample=1, model__verbose=0;, score=0.852 total time= 3.5s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.945 total time= 5.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.917 total time= 5.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 5.1s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 5.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 4.6s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.936 total time= 5.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 4.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.926 total time= 4.5s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 5.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=0.8, model__verbose=0;, score=0.880 total time= 4.4s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.899 total time= 3.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.872 total time= 3.4s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.870 total time= 3.8s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.907 total time= 4.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=1000, model__subsample=1, model__verbose=0;, score=0.852 total time= 3.9s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.936 total time= 6.1s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.927 total time= 6.0s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.926 total time= 6.3s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.935 total time= 6.1s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.6, model__verbose=0;, score=0.870 total time= 6.0s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.936 total time= 5.9s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.5s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.898 total time= 5.9s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.917 total time= 6.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=0.8, model__verbose=0;, score=0.870 total time= 5.8s [CV 1/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.890 total time= 5.0s [CV 2/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.881 total time= 4.6s [CV 3/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 4.9s [CV 4/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.898 total time= 5.0s [CV 5/5] END model__learning_rate=0.1, model__max_depth=6, model__n_estimators=2000, model__subsample=1, model__verbose=0;, score=0.852 total time= 5.6s
Out[148]:
GridSearchCV(cv=5,
estimator=Pipeline(steps=[('preprocessor',
ColumnTransformer(transformers=[('scaler',
StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),...
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01,
subsample=0.8,
verbose=1))]),
param_grid={'model__learning_rate': [0.001, 0.01, 0.1],
'model__max_depth': [3, 4, 5, 6],
'model__n_estimators': [100, 200, 500, 1000, 2000],
'model__subsample': [0.6, 0.8, 1],
'model__verbose': [0]},
scoring='accuracy', verbose=4)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
GridSearchCV(cv=5,
estimator=Pipeline(steps=[('preprocessor',
ColumnTransformer(transformers=[('scaler',
StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),...
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(learning_rate=0.01,
subsample=0.8,
verbose=1))]),
param_grid={'model__learning_rate': [0.001, 0.01, 0.1],
'model__max_depth': [3, 4, 5, 6],
'model__n_estimators': [100, 200, 500, 1000, 2000],
'model__subsample': [0.6, 0.8, 1],
'model__verbose': [0]},
scoring='accuracy', verbose=4)Pipeline(steps=[('preprocessor',
ColumnTransformer(transformers=[('scaler', StandardScaler(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY',
'indexInSequence']),
('yeoJohnson',
PowerTransformer(),
['relLeftShoulderX',
'relLeftShoulderY',
'relLeftElbowX',
'relLeftElbowY',
'relLeftWristX',
'relLeftWristY',
'relRightShoulderX',
'relRightShoulderY',
'relRightElbowX',
'relRightElbowY',
'relRightWristX',
'relRightWristY'])])),
('model',
GradientBoostingClassifier(max_depth=5, n_estimators=2000,
subsample=0.6))])ColumnTransformer(transformers=[('scaler', StandardScaler(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY',
'indexInSequence']),
('yeoJohnson', PowerTransformer(),
['relLeftShoulderX', 'relLeftShoulderY',
'relLeftElbowX', 'relLeftElbowY',
'relLeftWristX', 'relLeftWristY',
'relRightShoulderX', 'relRightShoulderY',
'relRightElbowX', 'relRightElbowY',
'relRightWristX', 'relRightWristY'])])['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY', 'indexInSequence']
StandardScaler()
['relLeftShoulderX', 'relLeftShoulderY', 'relLeftElbowX', 'relLeftElbowY', 'relLeftWristX', 'relLeftWristY', 'relRightShoulderX', 'relRightShoulderY', 'relRightElbowX', 'relRightElbowY', 'relRightWristX', 'relRightWristY']
PowerTransformer()
GradientBoostingClassifier(max_depth=5, n_estimators=2000, subsample=0.6)
In [149]:
gbc_gs_pipeline_model.best_score_
Out[149]:
0.9187903499830105
In [151]:
gbc_gs_pipeline_model.best_params_
Out[151]:
{'model__learning_rate': 0.1,
'model__max_depth': 5,
'model__n_estimators': 2000,
'model__subsample': 0.6,
'model__verbose': 0}
In [153]:
gbc_gs_pipeline_model.best_estimator_.score(X_test, y_test)
Out[153]:
0.9338235294117647
In [155]:
conf_matrix_plot = ConfusionMatrixDisplay.from_estimator(gbc_gs_pipeline_model.best_estimator_, X_test, y_test)
conf_matrix_plot.ax_.set_xticklabels(class_mapping.keys())
conf_matrix_plot.ax_.set_yticklabels(class_mapping.keys())
Out[155]:
[Text(0, 0, 'forehand'), Text(0, 1, 'backhand'), Text(0, 2, 'serve')]
In [184]:
feature_importance = gbc_gs_pipeline_model.best_estimator_["model"].feature_importances_
all_columns = np.array(all_input_channels + [f"{x}_transformed" for x in channels_to_power_transform])
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
# plt.subplot(1, 2, 2)
plt.figure(figsize=(8, 18))
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, all_columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Variable Importance')
plt.grid(True)
plt.show()
Zusammenfassung¶
Grundsätzlich ist es Möglich einen Gradient Boost Classifier auf Körperkoordinaten zu trainieren. Das zum schluss verwendete, komplette Datenset führte dazu, dass das Modell sehr gut generalisiert. Daher auch die auf dem Testset resultierende Accuracy von 91% (verglichen mit dem ersten Modell auf derselben Datenbasis von 90% auf dem Testset, sehr gut). Durch das aufwändige Hyper-Parameter-Tuning konnten lediglich noch 2% Accuracy herausgeholt werden. Auch bei diesem Modell stellte sich bei der Feature-Importance Analyse heraus, dass nach wie vor die Schulterpositionen als wichtigstes Feature hervorgehoben werden.
Offene Punkte¶
1. Genauere Evaluation des verwendeten Key-Point Estimators¶
Die Resultate der Key-Point estimation müssten viel genauer überprüft und untersucht werden.
2. Mehrere verschiedene Key-Point Estimators verwenden¶
Sehr spannend wäre, ein trainierter Classifier, der auf dem Output des einen Key-Point Estimators trainiert wurde, auf den Output eines anderen Key-Point Estimators (z.B. YOLO) zu validieren. In diesem Zusammenhang könnte man sogar auf denselben Bildern mehrere verschiedene Key-Point Estimator anwenden und mit allen Outputs ein einzelnes Klassifikationsmodell trainieren.
3. Anwendungsbeispiel¶
Ein Klassifikator, der nur anhand von Körperpositionen Tennis-Schläge klassifizieren kann, ist agnostisch gegenüber des verwendeten Key-Point Estimation Modells. Lediglich der Output des Key-Point Estimators müsste auf das richtige Input-Format gebracht werden. Viele dieser Pose Estimation Modelle sind so ausgelegt, dass diese schnell einzelne Video-Frames verarbeiten können. Soll nun zusätzlich noch ein Tennis-Schlag in einem Video klassifiziert werden, ist ein kompaktes, schnelles Modell sehr von Vorteil.